Intelligent spider for information retrieval to support mining-based price prediction for online auctioning

Since the emergence of online auctions in 1995, many individuals have joined auction markets. Sometimes, many bidders have made wrong decisions (e.g., ‘‘Winner Curse’’) due to their limited knowledge and resources. Unfortunately, search engines only show a list of search results to users, and fail to provide further analysis that could help improve users’ decision-making. To solve this problem, this study proposed an intelligent spider for information retrieval, and applied data mining technology to differentiate between customers. Two software programs, a URL searching agent and an auction data agent, are developed to automatically collect related information whenever users input the searched product. Two neural networks are used to perform data clustering and price prediction after this information is crawled and stored into a database. The first neural network adopts a self-organizing map (SOM) to cluster customer data into nine homogenous groups. The second backpropagation network (BPN) is then used to predict the final price. This study develops a prototype of the proposed spider, and conducts an empirical study by crawling over 1000 deals from Taiwan’s eBay. Finally, important information, such as predicted price, prediction error and historic records, are presented to the user. The user can thus easily target the right bidding policy for wining a bid based on the mining-based information about price prediction. 2006 Elsevier Ltd. All rights reserved.